What are "pivotal acts"?
A “pivotal act” typically refers to an act that uses powerful AI to take some unilateral action with the aim of significantly reducing existential risk
A risk of human extinction or the destruction of humanity’s long-term potential.
The term was originally coined by Eliezer Yudkowsky Co-founder of MIRI, known for his early pioneering work in AI alignment and his predictions that AI will probably cause human extinction.
Pivotal acts were proposed as a way for researchers to buy sufficient time to completely solve AI alignment
The problem of designing an AI to carry out a minimal pivotal act can be viewed as a limited formulation of the alignment problem: is it possible to give precise enough instructions to an AI powerful enough to do something (without unwanted side effects) which would actually prevent other people from deploying an unaligned AI?
When MIRI researchers talk about this problem, they often use the “strawberry task” as an example of the level of power needed for a pivotal act. The strawberry task involves producing two strawberries that are identical at the cellular level and then ceasing all action. If we had an alignment technique which could reliably get an AI to achieve this task with no unwanted side effects, then that AI could plausibly be used for a pivotal act.
The key here is that you want to build a system that is:
-
aligned so well that it does exactly what you want it to do;
-
aligned so well that it doesn't do anything you don't want it to do;
-
powerful enough to do something sufficiently complex to be impactful (but obviously not so powerful that alignment is intractable).
Pivotal acts are, by their nature, quite impactful and often outside of the overton window. Any process powerful enough to enact a pivotal act would be dangerous if unaligned.
For a critical view, Andrew Critch argues against this strategy of designing an AI to take a unilateral “pivotal act” since it will lead to distrust, increase conflict and fuel the race between different AI labs. He advocates for a collaborative pivotal process instead.